"Assessing Outcomes of an Undergraduate Archival Scholars Research Program"

Purpose and goals In 2016 University of Pittsburgh’s Library System (ULS) and Office for Undergraduate Research (OUR) partnered in launching Undergraduate Archival Scholar Research Program (ASRA). The program is intended to introduce undergraduate students to research. Students are awarded a $1,000 stipend to pursue an independent research project using ULS’s special and archival materials. Each student is assigned an academic mentor and a librarian or an archivist to support them in completing their projects. In the intervening years, 74 students completed the program. While we had anecdotal evidence about the success of the program, from participant feedback, it became clear that a more formalized approach of assessment was needed, in particular as we were increasing our fundraising efforts to support the program. In 2019, both ULS and OUR colleagues agreed on initiating work that would allow us to consider ASRA outcomes that both partners were interested in achieving, and creating mechanisms for assessing if these outcomes are met. Information gathered through the assessment could be used by both partners to advocate for continued funding for the program and/or identify areas in which the program may need to change to better meet its goals. Design, methodology, or approach To develop and map program outcomes, we used a logic model framework. It allowed us to think about the project outcomes in three stages: 1.immediate ones that focus on knowledge acquisition, 2. intermediate ones that deal with application of that knowledge, and finally, 3. long-term outcomes, that attempt to show if any material changes took place in program participants’ lives. Once these were agreed on and documented (in a formal logic model document), the ULS team began its work on considering best mechanisms for assessing if the program outcomes, as defined in the logic model, are being met. Since most of the program outcomes focused on the acquisition of research skills in general, and specifically, when working with primary materials, we developed an assessment rubric, based on the ACRL- RBMS/SAA guidelines. We used the rubric to design additional assessment tools including, pre- and post- participant surveys, prompts for participant research outputs, and an interview schedule for past participants. Findings The logic model proved a clear articulation of goals and outcomes of a multi-partner project. It allowed us to develop tools for assessment which can be consistently applied to measure project outcomes. The rubric proved to be invaluable in guiding the development of additional assessment mechanisms that are consistent with program outcome goals. Practical implications or value We believe that the logic model framework for articulating program outcomes and the rubric for operationalizing these outcomes can be applied successfully as constant assessment tools. In our project we used the ACRL-RBMS/SAA guidelines to develop our rubric, as ASRA focuses on research skills for work with primary sources. However, the rubric can be adapted to other types of literacies, based on the focus of the evaluated program or service